{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7323487e-d1d0-49e0-96c6-89ee9080c226",
   "metadata": {},
   "source": [
    "# Information extraction from Cheshire fire incident reports using Mistral language model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d64a9bc7-6512-4369-9f9c-681986a46672",
   "metadata": {},
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\">\n",
    "<ul class=\"toc-item\">\n",
    "<li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\">Introduction</a></span></li>\n",
    "<li><span><a href=\"#Prerequisites\" data-toc-modified-id=\"Prerequisites-2\">Prerequisites</a></span></li>\n",
    "<li><span><a href=\"#Imports\" data-toc-modified-id=\"Imports-3\">Imports</a></span></li>\n",
    "<li><span><a href=\"#Data-preparation\" data-toc-modified-id=\"Data-preparation-4\">Data preparation</a></span></li>\n",
    "<li><span><a href=\"#EntityRecognizer-model\" data-toc-modified-id=\"EntityRecognizer-model-5\">EntityRecognizer model</a></span></li>\n",
    "<ul class=\"toc-item\">\n",
    "<li><span><a href=\"#in-context-learning\" data-toc-modified-id=\"In-context-learning\">In-context learning</a></span>\n",
    "<li><span><a href=\"#Evaluate-model-performance\" data-toc-modified-id=\"Evaluate-model-performance-5.3\">Evaluate model performance</a></span>\n",
    "<li><span><a href=\"#Validate-results\" data-toc-modified-id=\"Validate-results-5.4\">Validate results</a></span></li>\n",
    "<li><span><a href=\"#Save-and-load-trained-models\" data-toc-modified-id=\"Save-and-load-trained-models-5.5\">Save and load trained models</a></span></li>\n",
    "</ul>\n",
    "<li><span><a href=\"#Model-inference\" data-toc-modified-id=\"Model-inference-6\">Model inference</a></span></li>\n",
    "<li><span><a href=\"#Publishing-the-results-as-feature-layer\" data-toc-modified-id=\"Publishing-the-results-as-feature-layer-7\">Publishing the results as feature layer</a></span></li>\n",
    "<li><span><a href=\"#Visualize-crime-incident-on-map\" data-toc-modified-id=\"Visualize-crime-incident-on-map- 8\">Visualize crime incident on map</a></span></li>\n",
    "<li><span><a href=\"#Create-a-hot-spot-map-of-crime-densities\" data-toc-modified-id=\"Create-a-hot-spot-map-of-crime-densities-9\">Create a hot spot map of crime densities</a></span></li>\n",
    "<li><span><a href=\"#Conclusion\" data-toc-modified-id=\"Conclusion-10\">Conclusion</a></span></li>\n",
    "<li><span><a href=\"#References\" data-toc-modified-id=\"References-11\">References</a></span></li>\n",
    "</ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cb55c43-2e4e-4210-b203-2e88c1a88222",
   "metadata": {},
   "source": [
    "## Introduction <a id=\"Introduction-1\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7768777-59ae-40ae-82ed-4ef4cebf015a",
   "metadata": {},
   "source": [
    "As text data continues to grow rapidly, extracting meaningful insights from large amounts of information is more important than ever. Large language models (LLMs) have emerged as powerful tools for processing unstructured data, significantly enhancing the accuracy and efficiency of information extraction. One of the key tasks that can be performed using large language models is entity extraction, which involves identifying and classifying entities—such as names, organizations, locations, dates, and other specific details—within a text.\n",
    "\n",
    "In this sample, we will explore how information extraction works using the Mistral language model in the `EntityRecognizer` class of the arcgis.learn API with the Cheshire fire incident reports dataset. The Cheshire fire dataset includes incident reports detailing fire incidents in Cheshire, covering information like locations, times, types of incidents, and response actions. This data can be valuable for analysis in understanding patterns, improving response strategies, and enhancing safety measures.\n",
    "\n",
    "Key entities to extract from fire incident reports include:\n",
    "- **Address**\n",
    "- **Date and Time**\n",
    "- **Incident Type**\n",
    "- **Number of Engines**\n",
    "- **Time Spent at Incident**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "700ee916-e7b7-4f54-a8c6-64377717c646",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "- Refer to the section **\"Install Deep Learning Dependencies of arcgis.learn Module\"** for detailed documentation on installing the dependencies: [Installation Guide](https://developers.arcgis.com/python/guide/install-and-set-up/#Install-deep-learning-dependencies).\n",
    "  \n",
    "- Follow these steps to download and install the Mistral model backbone:\n",
    "  \n",
    "  1. Download the [Mistral Model Backbone](https://esri.maps.arcgis.com/home/item.html?id=969d2fc57c834295af1a4a42cfd51f68).\n",
    "  2. Extract the downloaded zip file.\n",
    "  3. Open the Anaconda Prompt and navigate to the folder that contains `arcgis_mistral_backbone-1.0.0-py_0.tar.bz2`.\n",
    "  4. Run the following command:\n",
    "     - ```conda install --offline arcgis_mistral_backbone-1.0.0-py_0.tar.bz2```\n",
    "\n",
    "- To learn more about how `EntityRecognizer` works, please refer to the guide on [Named Entity Extraction Workflow with arcgis.learn](https://developers.arcgis.com/python/guide/how-named-entity-recognition-works/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac86ecca-e39a-43d4-a155-8b868e4edbad",
   "metadata": {},
   "source": [
    "## Necessary Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0b37eedb-e509-434f-b072-89f5bb7428fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import zipfile,unicodedata\n",
    "from itertools import repeat\n",
    "from pathlib import Path\n",
    "from arcgis.gis import GIS\n",
    "from arcgis.learn import prepare_textdata\n",
    "from arcgis.learn.text import EntityRecognizer\n",
    "from arcgis.geocoding import batch_geocode\n",
    "import re\n",
    "import os\n",
    "import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "48fc3c90-5295-4ee8-9538-1348ef690784",
   "metadata": {},
   "outputs": [],
   "source": [
    "gis = GIS('home')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b07c5348-4c42-4bc9-99eb-69a7d1de6cad",
   "metadata": {},
   "source": [
    "## Data preparation\n",
    "\n",
    "Data preparation involves splitting the data into training and validation sets, creating the necessary data structures for loading data into the model and so on.  The `prepare_data()` function can directly read the training samples in one of the above specified formats and automate the entire process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "03b906fd-6e9f-40d0-adea-bd407a54ced1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"item_container\" style=\"height: auto; overflow: hidden; border: 1px solid #cfcfcf; border-radius: 2px; background: #f6fafa; line-height: 1.21429em; padding: 10px;\">\n",
       "                    <div class=\"item_left\" style=\"width: 210px; float: left;\">\n",
       "                       <a href='https://ivt.maps.arcgis.com//home/item.html?id=ab3af7d1b8a24c1f8cc4e5bf3465d6bf' target='_blank'>\n",
       "                        <img src='' width='200' height='133' class=\"itemThumbnail\">\n",
       "                       </a>\n",
       "                    </div>\n",
       "\n",
       "                    <div class=\"item_right\"     style=\"float: none; width: auto; overflow: hidden;\">\n",
       "                        <a href='https://ivt.maps.arcgis.com//home/item.html?id=ab3af7d1b8a24c1f8cc4e5bf3465d6bf' target='_blank'><b>information_extraction_from_cheshire_fire_incident_reports_using_mistral_language_model</b>\n",
       "                        </a>\n",
       "                        <br/>information extraction from cheshire fire incident reports using mistral language model<br/><img src='https://ivt.maps.arcgis.com//home/js/jsapi/esri/css/images/item_type_icons/layers16.png' style=\"vertical-align:middle;\" width=16 height=16>Image Collection by api_data_owner\n",
       "                        <br/>Last Modified: October 16, 2024\n",
       "                        <br/>0 comments, 0 views\n",
       "                    </div>\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<Item title:\"information_extraction_from_cheshire_fire_incident_reports_using_mistral_language_model\" type:Image Collection owner:api_data_owner>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_data = gis.content.get('ab3af7d1b8a24c1f8cc4e5bf3465d6bf')\n",
    "training_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b6325d18-ea39-49a5-9494-ab73562367be",
   "metadata": {},
   "outputs": [],
   "source": [
    "filepath = training_data.download(file_name=training_data.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c7a7cda3-bcea-43a0-a323-5458b0e8d7f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "with zipfile.ZipFile(filepath, 'r') as zip_ref:\n",
    "    zip_ref.extractall(Path(filepath).parent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "43c1ec47-a880-4710-aab1-69621432f161",
   "metadata": {},
   "outputs": [],
   "source": [
    "json_path = Path(os.path.join(os.path.splitext(filepath)[0] , 'cheshire_fire_incident_reports.json'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "059f765d-07c6-4354-91dd-fe4d26f188ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.path.splitext(filepath)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c2d489ee-c80a-4957-9b81-fd199ca719b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = prepare_textdata(path= json_path, task=\"entity_recognition\", dataset_type='ner_json', class_mapping={'address_tag':'Address'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "698f8cab-2c5f-40e9-afc8-6df89923bf37",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>Address</th>\n",
       "      <th>Date_and_Time</th>\n",
       "      <th>Incident_Type</th>\n",
       "      <th>Number_of_Engines</th>\n",
       "      <th>Title</th>\n",
       "      <th>time_spent_at_incident</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Industrial paper shredder fire in Widnes Time ...</td>\n",
       "      <td>[Pickerings Road, Widnes]</td>\n",
       "      <td>[26/04/2018 - 21:07]</td>\n",
       "      <td>[fire]</td>\n",
       "      <td>[Two]</td>\n",
       "      <td>[Industrial paper shredder fire in Widnes]</td>\n",
       "      <td>[around half-an-hour]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Fire in field in Crewe Time of Incident: 07/03...</td>\n",
       "      <td>[Vicarage Road, Crewe]</td>\n",
       "      <td>[07/03/2018 - 12:48]</td>\n",
       "      <td>[fire]</td>\n",
       "      <td>[One]</td>\n",
       "      <td>[Fire in field in Crewe]</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Car fire in Chester Time of Incident: 05/02/20...</td>\n",
       "      <td>[Waverley Terrace, Chester]</td>\n",
       "      <td>[05/02/2018 - 22:26]</td>\n",
       "      <td>[car fire]</td>\n",
       "      <td>[One]</td>\n",
       "      <td>[Car fire in Chester]</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Industrial paper shredder fire in Widnes Time ...</td>\n",
       "      <td>[Pickerings Road, Widnes]</td>\n",
       "      <td>[26/04/2018 - 21:07]</td>\n",
       "      <td>[fire]</td>\n",
       "      <td>[Two]</td>\n",
       "      <td>[Industrial paper shredder fire in Widnes]</td>\n",
       "      <td>[around half-an-hour]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Van fire in Chester Time of Incident: 05/10/20...</td>\n",
       "      <td>[Christleton Road, Boughton, Chester]</td>\n",
       "      <td>[05/10/2018 - 10:45]</td>\n",
       "      <td>[van fire]</td>\n",
       "      <td>[One]</td>\n",
       "      <td>[Van fire in Chester]</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Industrial skip fire in Crewe Time of Incident...</td>\n",
       "      <td>[Gresty Road, Crewe]</td>\n",
       "      <td>[27/11/2018 - 09:16]</td>\n",
       "      <td>[fire]</td>\n",
       "      <td>[Two]</td>\n",
       "      <td>[Industrial skip fire in Crewe]</td>\n",
       "      <td>[45 minutes]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Skip fire in Warrington Time of Incident: 25/1...</td>\n",
       "      <td>[Griffiths Street, Warrington]</td>\n",
       "      <td>[25/10/2018 - 22:24]</td>\n",
       "      <td>[skip fire]</td>\n",
       "      <td>[One]</td>\n",
       "      <td>[Skip fire in Warrington]</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Fire at a dry cleaner's in Warrington Time of ...</td>\n",
       "      <td>[Gaskell Street, Warrington]</td>\n",
       "      <td>[03/12/2018 - 17:14]</td>\n",
       "      <td>[fire]</td>\n",
       "      <td>[One]</td>\n",
       "      <td>[Fire at a dry cleaner's in Warrington]</td>\n",
       "      <td>[45 minutes]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  \\\n",
       "0  Industrial paper shredder fire in Widnes Time ...   \n",
       "1  Fire in field in Crewe Time of Incident: 07/03...   \n",
       "2  Car fire in Chester Time of Incident: 05/02/20...   \n",
       "3  Industrial paper shredder fire in Widnes Time ...   \n",
       "4  Van fire in Chester Time of Incident: 05/10/20...   \n",
       "5  Industrial skip fire in Crewe Time of Incident...   \n",
       "6  Skip fire in Warrington Time of Incident: 25/1...   \n",
       "7  Fire at a dry cleaner's in Warrington Time of ...   \n",
       "\n",
       "                                 Address         Date_and_Time Incident_Type  \\\n",
       "0              [Pickerings Road, Widnes]  [26/04/2018 - 21:07]        [fire]   \n",
       "1                 [Vicarage Road, Crewe]  [07/03/2018 - 12:48]        [fire]   \n",
       "2            [Waverley Terrace, Chester]  [05/02/2018 - 22:26]    [car fire]   \n",
       "3              [Pickerings Road, Widnes]  [26/04/2018 - 21:07]        [fire]   \n",
       "4  [Christleton Road, Boughton, Chester]  [05/10/2018 - 10:45]    [van fire]   \n",
       "5                   [Gresty Road, Crewe]  [27/11/2018 - 09:16]        [fire]   \n",
       "6         [Griffiths Street, Warrington]  [25/10/2018 - 22:24]   [skip fire]   \n",
       "7           [Gaskell Street, Warrington]  [03/12/2018 - 17:14]        [fire]   \n",
       "\n",
       "  Number_of_Engines                                       Title  \\\n",
       "0             [Two]  [Industrial paper shredder fire in Widnes]   \n",
       "1             [One]                    [Fire in field in Crewe]   \n",
       "2             [One]                       [Car fire in Chester]   \n",
       "3             [Two]  [Industrial paper shredder fire in Widnes]   \n",
       "4             [One]                       [Van fire in Chester]   \n",
       "5             [Two]             [Industrial skip fire in Crewe]   \n",
       "6             [One]                   [Skip fire in Warrington]   \n",
       "7             [One]     [Fire at a dry cleaner's in Warrington]   \n",
       "\n",
       "  time_spent_at_incident  \n",
       "0  [around half-an-hour]  \n",
       "1                         \n",
       "2                         \n",
       "3  [around half-an-hour]  \n",
       "4                         \n",
       "5           [45 minutes]  \n",
       "6                         \n",
       "7           [45 minutes]  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c770b65-b9a9-426a-be0e-f57e101f01f1",
   "metadata": {},
   "source": [
    "## EntityRecognizer model\n",
    "\n",
    "`EntityRecognizer` model in `arcgis.learn` can be used with [Hugging Face Transformers](https://huggingface.co/transformers/v3.0.2/index.html) or with large language model backbones. For this sample use case we will use the Mistral model backbone to extract entities from the text.\n",
    "\n",
    "Run the command below to see what backbones are supported for the **entity recognition** task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fabc5720-98b4-4c9f-a51e-4968a7e50ebb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('mistral',)\n"
     ]
    }
   ],
   "source": [
    "print(EntityRecognizer.available_backbone_models(\"llm\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f36942bb-facd-4ecb-814e-26786f8cdcac",
   "metadata": {},
   "source": [
    "First we will create model using the `EntityRecognizer()` constructor and passing the following parameter:\n",
    "\n",
    "<code>data</code>: The databunch created using the <i>prepare_textdata</i> method.\n",
    "\n",
    "<code>backbone</code>: To use mistral as the model backbone, use <i>backbone=\"mistral\"</i>.\n",
    "\n",
    "<code>prompt</code>: Text string describing the task and its guardrails. This is an optional parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4894df87-6d82-4d0d-91e7-59699df8505c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9226d8cc45b045da9d41c36800b0703f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ner = EntityRecognizer(data,\n",
    "                       backbone=\"mistral\",\n",
    "                       prompt=\"Tag the input sentences in the named entity for the given classes, no other class should be tagged.\"\n",
    "                      )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "848bb3ac-ee27-4b61-a98a-e3f24be767ac",
   "metadata": {},
   "source": [
    "The Mistral model will automatically infer the classes from the dataset. The list of inferred class names is as follows:\n",
    "\n",
    "- Address  \n",
    "- Date_and_Time  \n",
    "- Incident_Type  \n",
    "- Number_of_Engines  \n",
    "- Title  \n",
    "- Time_spent_at_incident "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bea5234-52f3-43c3-9ff3-4575efd1f44b",
   "metadata": {},
   "source": [
    "## In-context learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a519c182-9fe5-4309-9879-3b54f233f641",
   "metadata": {},
   "source": [
    "The Mistral model utilizes in-context learning to generate predictions. Unlike traditional models that depend on lengthy training cycles, it can understand the task using just a few examples and a prompt. By incorporating this information into the input, the Mistral model gains a better understanding and can make more accurate predictions without needing retraining."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f55d6286-a63d-43e5-812a-0d322fb758fa",
   "metadata": {},
   "source": [
    "## Evaluate model performance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71b497c5-6687-4e66-808c-1b813868e9db",
   "metadata": {},
   "source": [
    "Important metrics to look at while measuring the performance of the `EntityRecognizer` model are **Precision**, **Recall** & **F1-measures** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "5f619dcf-1aee-43a0-894f-b5abd5a69e9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask is not set and cannot be inferred from input because pad token is same as eos token.As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.7"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ner.precision_score()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ac4d7c27-0e48-40a0-8b51-534b5c8e3466",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ner.recall_score()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "cb52f529-6cae-48f4-a696-45ed8907cb32",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.66"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ner.f1_score()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1336c9a-00ae-48ed-820f-814809563ef5",
   "metadata": {},
   "source": [
    "To find **precision**, **recall** & **f1** scores per label/class we will call the model's `metrics_per_label()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6e433fca-6cc3-4af4-bbc3-7a8f0e64a880",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Precision_score</th>\n",
       "      <th>Recall_score</th>\n",
       "      <th>F1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>time_spent_at_incident</th>\n",
       "      <td>0.17</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>address</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>incident_type</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.67</td>\n",
       "      <td>0.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>o</th>\n",
       "      <td>0.71</td>\n",
       "      <td>0.91</td>\n",
       "      <td>0.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date_and_time</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.90</td>\n",
       "      <td>0.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>number_of_engines</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.90</td>\n",
       "      <td>0.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.90</td>\n",
       "      <td>0.95</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        Precision_score  Recall_score  F1_score\n",
       "time_spent_at_incident             0.17          0.20      0.18\n",
       "address                            0.00          0.00      0.00\n",
       "incident_type                      1.00          0.67      0.80\n",
       "o                                  0.71          0.91      0.80\n",
       "date_and_time                      1.00          0.90      0.95\n",
       "number_of_engines                  1.00          0.90      0.95\n",
       "title                              1.00          0.90      0.95"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ner.metrics_per_label()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89f5410d-2e6b-47ed-95a4-f0867e32736a",
   "metadata": {},
   "source": [
    "## Validate results\n",
    "\n",
    "Now we have the trained model, let's look at how the model performs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "19353c75-3fa4-4a97-a727-8dbe24a75bff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TEXT</th>\n",
       "      <th>address</th>\n",
       "      <th>time_spent_at_incident</th>\n",
       "      <th>incident_type</th>\n",
       "      <th>title</th>\n",
       "      <th>number_of_engines</th>\n",
       "      <th>date_and_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Fire in the open at a farm in Chester Time of...</td>\n",
       "      <td>harthill lane, chester</td>\n",
       "      <td>1 hour and a half</td>\n",
       "      <td>fire in the open</td>\n",
       "      <td>fire in the open at a farm in chester</td>\n",
       "      <td>one</td>\n",
       "      <td>06/08/2018-14:29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Car fire in Northwich Time of Incident: 22/08...</td>\n",
       "      <td>walnut avenue, northwich</td>\n",
       "      <td></td>\n",
       "      <td>car fire</td>\n",
       "      <td>car fire in northwich</td>\n",
       "      <td>one</td>\n",
       "      <td>22/08/2018-20:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Grassland fire in Macclesfield Time of Incide...</td>\n",
       "      <td>hooleyhey lane, macclesfield</td>\n",
       "      <td></td>\n",
       "      <td>grassland fire</td>\n",
       "      <td>grassland fire in macclesfield</td>\n",
       "      <td>one</td>\n",
       "      <td>02/07/2018-13:55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Tree fire, Warrington Time of Incident: 24/02...</td>\n",
       "      <td>off shackleton close, warrington</td>\n",
       "      <td></td>\n",
       "      <td>tree fire</td>\n",
       "      <td>tree fire, warrington</td>\n",
       "      <td>one</td>\n",
       "      <td>24/02/2018-14:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Explosion at a house in Warrington Time of In...</td>\n",
       "      <td>heath lane, warrington</td>\n",
       "      <td>2 hours 40 minutes</td>\n",
       "      <td>explosion</td>\n",
       "      <td>explosion at a house in warrington</td>\n",
       "      <td>three</td>\n",
       "      <td>28/09/2018-15:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Digger fire in Warrington Time of Incident: 0...</td>\n",
       "      <td>manchester road, warrington</td>\n",
       "      <td></td>\n",
       "      <td>digger fire</td>\n",
       "      <td>digger fire in warrington</td>\n",
       "      <td>two</td>\n",
       "      <td>07/12/2018-20:19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Small fire in a garden in Warrington Time of ...</td>\n",
       "      <td>waterside, warrington</td>\n",
       "      <td>around half-an-hour</td>\n",
       "      <td>fire</td>\n",
       "      <td>small fire in a garden in warrington</td>\n",
       "      <td>one</td>\n",
       "      <td>05/08/2018 - 18:14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Fire involving rubbish in Warrington Time of ...</td>\n",
       "      <td>bridge lane, warrington</td>\n",
       "      <td>around 20 minutes</td>\n",
       "      <td>fire</td>\n",
       "      <td>fire involving rubbish in warrington</td>\n",
       "      <td>one</td>\n",
       "      <td>18/05/2018-08:29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Small fire in the open in Warrington Time of ...</td>\n",
       "      <td>parkfields lane, poulton-with-fearnhead, warri...</td>\n",
       "      <td>20 minutes</td>\n",
       "      <td>fire</td>\n",
       "      <td>small fire in the open in warrington</td>\n",
       "      <td>one</td>\n",
       "      <td>25/07/2018-19:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Greenhouse fire in Widnes Time of Incident: 1...</td>\n",
       "      <td>scott avenue, widnes</td>\n",
       "      <td></td>\n",
       "      <td>greenhouse fire</td>\n",
       "      <td>greenhouse fire in widnes</td>\n",
       "      <td>one</td>\n",
       "      <td>10/05/2018-18:52</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                TEXT  \\\n",
       "0   Fire in the open at a farm in Chester Time of...   \n",
       "1   Car fire in Northwich Time of Incident: 22/08...   \n",
       "2   Grassland fire in Macclesfield Time of Incide...   \n",
       "3   Tree fire, Warrington Time of Incident: 24/02...   \n",
       "4   Explosion at a house in Warrington Time of In...   \n",
       "5   Digger fire in Warrington Time of Incident: 0...   \n",
       "6   Small fire in a garden in Warrington Time of ...   \n",
       "7   Fire involving rubbish in Warrington Time of ...   \n",
       "8   Small fire in the open in Warrington Time of ...   \n",
       "9   Greenhouse fire in Widnes Time of Incident: 1...   \n",
       "\n",
       "                                             address time_spent_at_incident  \\\n",
       "0                             harthill lane, chester      1 hour and a half   \n",
       "1                           walnut avenue, northwich                          \n",
       "2                       hooleyhey lane, macclesfield                          \n",
       "3                   off shackleton close, warrington                          \n",
       "4                             heath lane, warrington     2 hours 40 minutes   \n",
       "5                        manchester road, warrington                          \n",
       "6                              waterside, warrington    around half-an-hour   \n",
       "7                            bridge lane, warrington      around 20 minutes   \n",
       "8  parkfields lane, poulton-with-fearnhead, warri...             20 minutes   \n",
       "9                               scott avenue, widnes                          \n",
       "\n",
       "      incident_type                                  title number_of_engines  \\\n",
       "0  fire in the open  fire in the open at a farm in chester               one   \n",
       "1          car fire                  car fire in northwich               one   \n",
       "2    grassland fire         grassland fire in macclesfield               one   \n",
       "3         tree fire                  tree fire, warrington               one   \n",
       "4         explosion     explosion at a house in warrington             three   \n",
       "5       digger fire              digger fire in warrington               two   \n",
       "6              fire   small fire in a garden in warrington               one   \n",
       "7              fire   fire involving rubbish in warrington               one   \n",
       "8              fire   small fire in the open in warrington               one   \n",
       "9   greenhouse fire              greenhouse fire in widnes               one   \n",
       "\n",
       "        date_and_time  \n",
       "0    06/08/2018-14:29  \n",
       "1    22/08/2018-20:39  \n",
       "2    02/07/2018-13:55  \n",
       "3    24/02/2018-14:38  \n",
       "4    28/09/2018-15:01  \n",
       "5    07/12/2018-20:19  \n",
       "6  05/08/2018 - 18:14  \n",
       "7    18/05/2018-08:29  \n",
       "8    25/07/2018-19:01  \n",
       "9    10/05/2018-18:52  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ner.show_results()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "895c8b02-93e9-4117-b7f5-ac9dff3575a3",
   "metadata": {},
   "source": [
    "## Save and load trained models\n",
    "\n",
    "Once you are satisfied with the model, you can save it using the `save()` method. This creates an Esri Model Definition (EMD file) that can be used for inferencing on new data. \n",
    "Saved models can also be loaded back using the `load()` method. `load()` method takes the path to the emd file as a required argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8c48d442-f2c4-453c-a588-215c1d11dc13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing model metrics...\n",
      "Model has been saved to C:\\Users\\sur11226\\AppData\\Local\\Temp\\information_extraction_from_cheshire_fire_incident_reports_using_mistral_language_model\\models\\cheshire_fire_model\n"
     ]
    }
   ],
   "source": [
    "ner.save('cheshire_fire_model')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff76446f-1ec3-4dbe-8196-4bad5ef56f57",
   "metadata": {},
   "source": [
    "## Model Inference\n",
    "\n",
    "Now we can use the trained model to extract entities from new text documents using `extract_entities()` method. This method expects the folder path of where new text document are located, or a list of text documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "16d5b684-1caa-47cf-b14f-4caf5e73b798",
   "metadata": {},
   "outputs": [],
   "source": [
    "reports = os.path.join(os.path.splitext(filepath)[0] , 'reports')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8ee5e392-3238-4a46-bc19-9d4574550103",
   "metadata": {},
   "outputs": [],
   "source": [
    "item_names = os.listdir(reports)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b43d8cd7-d05f-40fa-944d-935070178cde",
   "metadata": {},
   "outputs": [],
   "source": [
    "# item_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "3d38046c-c034-4c84-a4dd-374fccab85ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "reports_list = []\n",
    "file_encoding=\"utf-8\"\n",
    "for filename in item_names:\n",
    "    file_path = os.path.join(reports, filename)\n",
    "    ext = os.path.splitext(filename)[-1].lower().replace(\".\", \"\")\n",
    "    if ext == \"txt\":\n",
    "        with open(file_path, \"r\", encoding=file_encoding, errors=\"ignore\") as f:\n",
    "            reports_list.append(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "851724f1-8578-419c-a85a-28ec2eb133bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='25' class='' max='25' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [25/25 04:37<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "results = ner.extract_entities(reports_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "f86141ae-6a66-4fae-9164-0ee2ddb0de86",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TEXT</th>\n",
       "      <th>address</th>\n",
       "      <th>time_spent_at_incident</th>\n",
       "      <th>incident_type</th>\n",
       "      <th>title</th>\n",
       "      <th>number_of_engines</th>\n",
       "      <th>date_and_time</th>\n",
       "      <th>Filename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Person stuck in lift in Neston Time of Inciden...</td>\n",
       "      <td>brook street, neston</td>\n",
       "      <td></td>\n",
       "      <td>person in lift</td>\n",
       "      <td>person stuck in lift in neston</td>\n",
       "      <td>one</td>\n",
       "      <td>30/01/2015 - 16:36</td>\n",
       "      <td>Example_0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Arson investigation after fire in Crewe Time o...</td>\n",
       "      <td>charlesworth street, crewe</td>\n",
       "      <td></td>\n",
       "      <td>fire</td>\n",
       "      <td>arson investigation after fire in crewe</td>\n",
       "      <td>two</td>\n",
       "      <td>28/01/2015-23:09</td>\n",
       "      <td>Example_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Grill pan fire in Macclesfield Time of Inciden...</td>\n",
       "      <td>buxton road, macclesford</td>\n",
       "      <td></td>\n",
       "      <td>grill pan fire</td>\n",
       "      <td>grill pan fire in macclesfield</td>\n",
       "      <td>one</td>\n",
       "      <td>03/01/2015 - 13:07</td>\n",
       "      <td>Example_2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Collision involving car and HGV on M6 Time of ...</td>\n",
       "      <td>m6 north, j20</td>\n",
       "      <td></td>\n",
       "      <td>collision</td>\n",
       "      <td>collision involving car and hgv on m6</td>\n",
       "      <td>two,one</td>\n",
       "      <td>28/01/2015-22:49</td>\n",
       "      <td>Example_3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Road traffic collision in Macclesfield Time of...</td>\n",
       "      <td>congleton road, gawsworth, macclesfield</td>\n",
       "      <td>n/a</td>\n",
       "      <td>road traffic collision</td>\n",
       "      <td>road traffic collision in macclesfield</td>\n",
       "      <td>one,one</td>\n",
       "      <td>28/01/2015-17:36</td>\n",
       "      <td>Example_4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                TEXT  \\\n",
       "0  Person stuck in lift in Neston Time of Inciden...   \n",
       "1  Arson investigation after fire in Crewe Time o...   \n",
       "2  Grill pan fire in Macclesfield Time of Inciden...   \n",
       "3  Collision involving car and HGV on M6 Time of ...   \n",
       "4  Road traffic collision in Macclesfield Time of...   \n",
       "\n",
       "                                   address time_spent_at_incident  \\\n",
       "0                     brook street, neston                          \n",
       "1               charlesworth street, crewe                          \n",
       "2                 buxton road, macclesford                          \n",
       "3                            m6 north, j20                          \n",
       "4  congleton road, gawsworth, macclesfield                    n/a   \n",
       "\n",
       "            incident_type                                    title  \\\n",
       "0          person in lift           person stuck in lift in neston   \n",
       "1                    fire  arson investigation after fire in crewe   \n",
       "2          grill pan fire           grill pan fire in macclesfield   \n",
       "3               collision    collision involving car and hgv on m6   \n",
       "4  road traffic collision   road traffic collision in macclesfield   \n",
       "\n",
       "  number_of_engines       date_and_time   Filename  \n",
       "0               one  30/01/2015 - 16:36  Example_0  \n",
       "1               two    28/01/2015-23:09  Example_1  \n",
       "2               one  03/01/2015 - 13:07  Example_2  \n",
       "3           two,one    28/01/2015-22:49  Example_3  \n",
       "4           one,one    28/01/2015-17:36  Example_4  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7360d29f-de9d-499f-b836-40d2d14f0eaf",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "This sample demonstrates how `EntityRecognizer()` from `arcgis.learn` can be used for information extraction from Cheshire fie incident reports using the Mistral large language model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8e1ef1d-60c9-4d42-b99e-5bcc07be6092",
   "metadata": {},
   "source": [
    "## References\n",
    "Mistral-7B HuggingFace: <a href=\"https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2\">https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2</a><br/>\n",
    "Mistral-7B MistralAI: <a href=\"https://mistral.ai/news/announcing-mistral-7b/\">https://mistral.ai/news/announcing-mistral-7b</a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:conda-arcgis_13_DEC_24] *",
   "language": "python",
   "name": "conda-env-conda-arcgis_13_DEC_24-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
